Problem

Heroku Connect was leaving Contact_Editor_Affiliation__c rows permanently FAILED in the mirror with errors like:

Foreign key external ID: <school-uuid> not found for field EditorUUID__c in entity Editor__c
Foreign key external ID: <user-uuid>   not found for field Pi_Accounts_Unique_ID__c in entity Contact

This happens because School and Role after_commit callbacks both fire in the same transaction and enqueue independent jobs (SchoolSyncJob and RoleSyncJob) on the salesforce_sync queue. The jobs use different concurrency keys, so the role write to the Heroku Connect mirror can be pushed to Salesforce before the Editor record lands. When that happens the lookup-by-external-id fails, Salesforce rejects the INSERT, and Heroku Connect does not auto-retry foreign-key resolution failures — the mirror row stays FAILED forever.

The same pattern happens against Contact when a role is created before the upstream Pi Accounts → Salesforce Contact pipeline has materialised the user.

Evidence from production

Investigating Editors in Salesforce with no Contact_Editor_Affiliation__c:

• 1,114 Editors with no CEA, of which 1,098 are rejected schools (working as designed).
• 16 active residual cases. After classifying via _hc_err:
  • 9 schools had no mirror row at all (one-off backfill recovered them).
  • 1 school (Gurnard, created 3 days ago) had _hc_err = "Foreign key external ID … not found for field EditorUUID__c in entity Editor__c". The parent Editor__c had since synced (sfid populated, _hc_lastop = SYNCED), so the affiliation INSERT would now succeed — but Connect never retries.
  • 3 schools failed with the equivalent Contact error and have no Salesforce::Contact mirror row at all (separate upstream issue, not in scope).
• A new occurrence of the Editor race appeared in the Heroku Connect log at 00:01:04 UTC today for school 6a228ff8-…, confirming the race is still actively breaking new affiliations.

The 10 recoverable cases were re-synced manually; the SOQL count for not-rejected Editors with no CEA dropped 16 → 6 (3 missing-Contact + 3 by-design schools with only student / no roles).

Fix

Before writing to the affiliation mirror, verify the parent records exist in the local Heroku Connect mirror with sfid populated (i.e. Salesforce has acknowledged them). If either parent isn't ready, raise SalesforceRecordNotFound.

SalesforceSyncJob already declares:

retry_on SalesforceRecordNotFound, wait: :polynomially_longer, attempts: 10

…so this slots into the existing retry mechanism (same one ContactSyncJob already relies on) without any new error class, queue, or backoff config.

Behaviour

What this does not fix

• 1,098 rejected schools without affiliations — working as designed.
• 3 schools whose users (1 live, 2 discarded/invalidated) have no Contact in Salesforce — needs investigation upstream of editor-api.
• Existing FAILED mirror rows already in production — those need a one-off cleanup (delete sfid: nil rows then re-enqueue RoleSyncJob); already done for the 10 recoverable cases above.

Tests

spec/jobs/salesforce/role_sync_job_spec.rb:

• Existing happy-path / save-failure / student-role / feature-flag-off contexts retained, with parent rows now provisioned via let! so the new guards pass.
• New contexts:
  • Salesforce::School exists with sfid: nil → raises SalesforceRecordNotFound (Editor not yet synced)
  • No Salesforce::School row at all → raises SalesforceRecordNotFound
  • Salesforce::Contact exists with sfid: nil → raises SalesforceRecordNotFound (Contact not yet synced)
  • No Salesforce::Contact row at all → raises SalesforceRecordNotFound
  • In all failure cases, no row is written to the affiliation mirror.

Test plan

• CI rspec passes
• Rubocop passes (verified locally)
• After deploy, monitor Heroku Connect error log for Foreign key external ID … not found errors on contact_editor_affiliation__c — should drop to ~0 for new schools
• Spot-check a freshly created school: confirm Contact_Editor_Affiliation__c rows appear in Salesforce within retry window

Made with Cursor